Methods and systems for distinguishing point sources

ABSTRACT

An optical smart sensor combines a phase grating with a rolling shutting to distinguish between modulated point sources. Employing a phase grating in lieu of a lens dramatically reduces size and cost, while using timing information inherent to imaging techniques that used a rolling shutter allows the smart sensor to distinguish point sources quickly and easily using a single frame of image data.

BACKGROUND

Many applications for computer vision involve locating, tracking, anddistinguishing between point sources. Established tracking solutionsoften use custom passive or active markers. For example, avirtual-reality controller glove can present multiple markers to revealits orientation to an image sensor. If one of the markers becomesoccluded, it is useful to know which one. Sources can be flashed indifferent patterns and monitored frame-to-frame for identification, andpotentially many frames may be required after rapid movement forconfident disambiguation. Imaging systems that do this well can be bulkyand expensive.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is illustrated by way of example, and not byway of limitation, in the figures of the accompanying drawings and inwhich like reference numerals refer to similar elements and in which:

FIG. 1A depicts an optical system 100 with an optical smart sensor 103that combines a phase grating with a rolling shutter to distinguishbetween modulated point sources.

FIG. 1B is a simplified view of system 100 illustrating how the varyingintensities of illumination sources LED1 and LED2 interact with therolling shutter to produce distinct PSFs 160 and 165 on an array 107.

FIG. 1C is a waveform diagram 175 illustrating operational timing ofoptical system 100 of FIGS. 1A and 1B.

FIG. 2 depicts image data from an optical system in which a 20 mW, 850nm LED illuminates an image sensor with a 1 mm aperture at a range of 1m.

FIG. 3A is a diagram 300 relating SNR to modulation frequency for asystem in which the exposure period is set to 1 ms and the frequency ofa single-sinusoid modulation function is varied over a range of fromzero to about 10 kHz.

FIG. 3B is a diagram 310 illustrating SNR as a function of exposure timefor a single point-source response (PSR) modulated using 3 kHz and 4.25kHz sinusoids.

FIG. 4A is a cut-away view of an imaging image sensor 100 of FIG. 1A,like-identified elements being the same or similar.

FIG. 4B is a plan view of imaging image sensor 100 of FIG. 1A inaccordance with an embodiment in which grating 105 includes spiralfeatures 430 and 435 to produce two-dimensional diffraction patterns.

FIG. 5 illustrates a system 500 in which a smart image sensor 505monitors the locations of some number of autonomous aerial vehicles(UAVs) 510 that emit from uniquely modulated illumination sources.

DETAILED DESCRIPTION

FIG. 1A depicts an optical system 100 with an optical smart sensor 103that combines a dispersive optic, such as a phase grating, with arolling shutter to distinguish between modulated point sources, a pairof light-emitting diodes LED1 and LED2 in this example. Employing aphase grating in lieu of a lens dramatically can reduce size and cost,while using timing information inherent to a rolling shutter allowssystem 100 to distinguish point sources quickly and easily and, in someembodiments, can enable such distinguishing using only a single frame ofimage data. As used herein, the term “frame” refers to a singleelectronically coded image, and is not limited to video technology.

Smart optical sensor 103 includes a phase grating 105 that exhibits anearly invertible point-spread function (PSF) to produce a diffractiveresponse to point sources LED1 and LED2 on an underlying array 107 ofpixels 110. Array 107 can be part of a CMOS image sensor with rows andcolumns of pixels 110 under control of a microprocessor 115 via anaddress generator 120 and a row decoder 125. Microprocessor 115 readsfrom successive rows of pixels 110, from top to bottom and left toright, using a column scanner 130 and a sample-and-holdanalog-to-digital converter (ADC) 135. Array 107 includes only sixteenpixels 110 for ease of illustration, though inexpensive arrays forsensing visible light commonly include a million or more.

Address generator 120 is a shift register that sequentially scans all ofthe rows and generates row-reset (RST1-RST4) and row-select (SEL1-SEL4)signals for row address decoder 125. Row decoder 125 applies thesesignals to control the exposure for each row of pixels. In particular,each row of pixels becomes photosensitive upon receiving a row-resetsignal RST # and stops collecting photons upon receiving a row-selectsignal SEL #. ADC 135 reads out each row of sample values thus collectedone column at a time, from left to right. There is but one column lineCOL # per column so readout timings are different for each row, witheach successive row delayed from the last by a row time RT. The rows arethus exposed at slightly different times. In this example, the lower arow in a captured frame the later the observation of the imaged scene.This approach is commonly referred to as “rolling shutter.”

Sensors that employ rolling-shutter timing can produce objectionableimage artifacts in still or video frames. For example, samplingdifferent portions of a moving object over time produces non-rigiddeformations that can be very distracting, and light sources that varyin intensity over the timing of a frame can show as horizontal brightand dark bands on the captured image. System 100 takes advantage of thislatter form of distortion and attributes of the PSF of grating 105 todistinguish between illumination sources.

Modulators 136 and 137 modulate power to respective illumination sourcesLED1 and LED2 so that the output intensities of LED1 and LED2 varysinusoidally over respective modulation periods MT1 and MT2. Themodulated intensity combines with the offset row timings of the rollingshutter to superimpose relatively dark and bright horizontal bands onsampled point-spread responses of the captured image. The observedbanding is a function of the modulation period of the sampled light, andthus can be used to distinguish between point-spread responses (PSRs),and thus between the modulated illumination sources.

In the embodiment of FIG. 1, a memory 140 within optical smart sensor103 stores a kernel 145 that represents the PSF of grating 105, possiblyin combination with the underlying array 107, and modulation signatures150 and 155 that characterize the modulations applied by modulators 136and 137, and thus distinguish illumination sources LED1 and LED2.Processor 115 can use kernel 145 locate PSRs in a captured frame orframes and extract intensity information from the PSRs that can bedemodulated and matched against modulation signatures 150 and 155 toidentify the corresponding point sources. Other point sources in thescene, potentially with different modulations and therefore conveyingdifferent information, may be present and independently demodulatedwithout cross interference. Modulation signatures 150 and 155 may simplyreflect respective modulation frequencies, though more complexmodulation schemes may be used.

FIG. 1B is a simplified view of system 100 illustrating how the varyingintensities of illumination sources LED1 and LED2 interact with therolling shutter to produce distinct PSRs 160 and 165 on array 107.Optical smart sensor 103 can calculate the angular positions of pointsources by noting the locations of their PSRs on array 107, and candistinguish between modulated point sources by considering the stripingof each PSF. Optical smart sensor 103 can thus determine the positionsof multiple point sources using a single frame of image data. Anunmodulated point-spread response (PSR) 170 is depicted at the bottom ofFIG. 1B for comparison. As compared with a lens, the PSR of which is aspot, PSR 170 of grating 105 is spread out over more rows of pixels 110.In one embodiment, for example the diameter of PSR 170 extendsvertically across about sixty rows of pixels. Grating 105 produces amulti-armed spiral PSF in this embodiment, though other shapes can beused.

PSR 170 represents a sharply focused diffractive response from anexemplary imaged point source as it may appear at array 107. PSR 170 isillustrated as dark on a light background for ease of illustration, butwould appear as a relatively bright pattern on a darker background. PSR170 illuminates a set of R pixels within a convex hull 173, the smallestconvex set of S pixels that encompasses all the illuminated pixels.(Convex hull 173 may be visualized as the shape formed by a rubber bandstretched around PSR 170.) To find the convex hull for a given imagingdevice, PSR 170 can be sampled by array 107. With the brightest pixel(s)serving as a reference, those pixels with at least 10% of that maximumbrightness are included in the set of R pixel values representative ofthe response. Convex hull 173 is the smallest convex set of pixels 110that includes that set of R pixel values. In this example, PSR 170illuminates a pattern such that the set of illuminated pixels R is lessthan half of the convex set S (S>2R). The convex hull is not used forimage acquisition or analysis, but affords a measure of response areathat can be used to characterize the ratio of active pixels relative toa PSR and the richness of spatial modulations spread out over an areagreater than typical of focused or defocused conventional optics. Theset of spatial modulations within hull 173 allow processor 115 toprecisely locate the center of PSR 170, increase motion sensitivity, andextend over many rows of pixels to support point-source discriminationin the manner detailed herein.

FIG. 1C is a waveform diagram 175 illustrating operational timing ofoptical system 100 of FIGS. 1A and 1B. A waveform for each row shows theexposure timing, which is to say the duration that each row is exposedto light from an imaged scene. Sampled intensity data is transferred outand pixels in each row are reset between exposures. These operations arewell known so the details are omitted.

Considering illumination source LED2, the exposure times for successiverows are offset by row time RT so that the pixels 110 in each rowintegrate the modulated intensity over different ranges of intensity.The resultant impact on row intensities produces the striping of the PSR165 associated with LED2. The modulation period MT1 of LED1 likewiseproduces a striping in PSR 160. However, the spacings between thestripes in PSRs 160 and 165 are a function of their respectivemodulation periods MT1 and MT2, and can thus be used to distinguishbetween illumination sources LED1 and LED2. Both modulation periods MT1and MT2 are greater than row time RT (the timing delay betweensuccessive row exposures) and less than a frame time FT (the cumulativetime of all row exposures for a single image frame).

FIG. 2 depicts image data from an optical system in which a 20 mW, 850nm LED illuminates an image sensor with a 1 mm aperture at a range of 1m. Light was collected using a four-megapixel (1688×1520 pixels) CMOSimage sensor at thirty frames per second. At left is a 60×60 pixelsub-image cropped from full-resolution image data and centered on a PSR200 for an unmodulated illumination source. An accompanying intensitygraph 205 shows the accumulated intensity value of each of the sixtyrows normalized for the PSF of grating 107 as a function of pixel row.The line at the top of graph 205 shows that each row is essentially asbright as it should be to describe the corresponding optical slice ofunmodulated PSR 200. The flat response identifies the source asunmodulated.

With reference to FIG. 1A, imaging device 103 is programmed to locatePSR 200 amidst a four-megapixel frame, extract the row-intensity datadepicted in graph 205, and extract modulation information from therow-intensity data. To locate each PSR in a given frame of a sampledinterference pattern, processor 115 correlating the sampled interferencepattern with kernel 145. Peak responses produced by this correlationrepresent matches between PSRs and the calibrated PSF represented bykernel 145. Ideally, one could reconstruct an image of the capturedscene by inverting the effect of the PSF of grating 105 on theinterference pattern using linear algebra. In practice, however, the PSFis not well conditioned and the interference pattern is noisy. Applyinga regularized pseudoinverse to the interference pattern is thus morepractical. One popular pseudoinverse is given by Tikhonovregularization, which is well known to those of skill in the art. Thisreconstruction process can be accomplished using low-resolution imagedata to save time and power.

The image reconstructed from the sampled interference pattern exhibits abright spot for each PSR, each spot indicative of the center of thecorresponding PSR. Processor 115 crops the captured interference patternusing a window centered on each PSR location and sized to just encompassthe PSR, PSR 200 cropped within a 60×60 pixel window in the instantexample. Processor 115 then extracts row-intensity data from the croppedPSR. In one embodiment processor 115 accumulates each row of intensitiesin each cropped PSR using a function Demodulate( ) that takes a cropped60×60 sub-image Crop (e.g., PSR 200) and the PSF of grating 105,represented by kernel 145, and returns a one-dimensional signal in whicheach element is an estimate of the average intensity of the point lightsource during the integration interval of the corresponding row of theinput image, given the same row of the PSF of grating 105. The resultfor PSR 200 is the row-intensity data depicted in graph 205.

Assuming a function SumRows( ), which takes as input an N×M image andreturns an M-element signal whose each value is the sum of the N pixelson the corresponding row, function Demodulate( ) can be expressedmathematically as SumRows(Crop*PSF)/(gamma+SumRows(PSF*PSF)). As gammagoes to zero, the function Demodulate( ) tends toward taking the croppedinterference pattern Crop and dividing by the PSF of grating 105. Sincethere are rows where the PSF is not as strong as in other rows, and atthe top and bottom of the cropped capture PSF it does actually taper tozero, we do not simply divide by zero so as to avoid applying a largegain to row sum values that are largely due to noise. Gamma in this caseensures that as the PSF tapers to zero, so does the demodulation output.In rows where the PSF is strong, function Demodulate( ) does somethingvery close to dividing by the appropriate component of the PSF and thusprovides a flat, unbiased estimate of the corresponding source intensitywith the effect of the PSF canceled out.

At center of FIG. 2 is a PSR 210 for the same point source used togenerate unmodulated PSR 200 but modulated 100% (dark to bright) at 1.5kHz. PSR 210 is clearly identifiable as a point-source response and canbe used to locate the corresponding point source. The accompanyingintensity graph 215, showing the intensity normalized for the gratingPSF as a function of pixel row, reflects the modulation period over therange of rows. On the right, a PSR 220 for the same point source used togenerate PSFs 200 and 210 modulated 100% at 2.5 kHz appears much likethe other PSFs but is easily distinguished by the accompanying intensitygraph 225. Other point sources in the scene, potentially with differentmodulations (and therefore conveying different information) may bepresent and independently demodulated without cross interference.Processor 115 demodulates the collections of intensity valuesrepresented by graphs 205, 215, and 225 to extract modulationcharacteristics unique to each point source. In this way, optical system100 can distinguish between the point sources responsible for PSRs 200,210, and 220.

The ability to distinguish point sources confers a degree of “jamresistance,” where a receiver is able to perform demodulation on onlythe pixels that are expected to be influenced by a desired point source.Even an extremely bright (bright enough to cause saturation of thepixels that see it) light source displaced from the point source ofinterest can be ignored.

One simple application for this concept is in 3D position and poseestimation for virtual-reality (VR) applications. A VR helmet may havean array of point sources on it, observed by a camera fixed to a basestation. If the point sources are modulated differently (e.g., simplesinusoids repeating unconditionally) they can be distinguished in asingle frame. An unambiguous orientation can be derived for the helmetwithout any potentially unreliable disambiguation algorithm that mayrequire extensive temporal history. Frames can be combined in otherembodiments, such as to extend the discernable modulation periods.

In a VR headset, the LEDs of different parts of the wearable gear(including headsets, gloves, etc., of many users) each could bedistinguished on a per-frame basis. Other game controllers such as“magic wands” similarly could link an object's digital identity with itslocation in space given only the ability to modulate luminosity (or evenmerely reflectivity).

In an Internet-of-Things (IoT) application, an array of sensors mayrequire very-low-power, one-way communication to a mains-powered basestation. Each sensor may run on harvested energy and only infrequentlyilluminate a modulated LED to transmit a sensor reading back to base.This may include smart building applications where employees orcustomers wear low-power tags that periodically broadcast a unique IDand/or very-low-bandwidth sensor data. In a smart warehouse, a shippingcontainer may report internal temperature measurements, etc. A gamingapplication may give each player a simple, inexpensive controller withonly a single IR LED. One or more smart optical sensors viewing theplaying area would be able to locate and receive control inputs fromeach player (e.g., laser-tag participant). There may also beapplications that overlap with current near-field-communication usecases, for example transmitting a personal identification number forsecure and convenient pairing between devices.

In the IoT sensor example, a variety of modulation schemes may beapplicable, including pulse position modulation, orthogonal frequencydivision multiplexing, etc. In some embodiments, only amplitude isstraightforward to demodulate as the phase of modulation in the imagewill vary arbitrarily. Some embodiments include an intra-periodsynchronization mechanism to make phase available for modulation. Themodulation task is made easier if the capture parameters of the sensorare known. If the transmitter and receiver are not co-designed, thereceiver may be able to change its frame rate, exposure etc. adaptivelyto optimize reception from the transmitter.

With a fixed exposure time, certain modulation frequencies will not betransmitted through to a rolling-shutter image. If the modulationfrequency is an integer multiple of the reciprocal of the exposure time,the modulation can be canceled and not be reflected in the image.Capturing frames at two different exposures will make those frequenciesobservable, potentially allowing the two frames to be combined to derivea single spectrum with no zeroes and more available bandwidth. Somerolling-shutter image sensors provide a mechanism to automaticallyswitch between two or more exposures on consecutive frames, which may beuseful in this approach.

In addition to the integer-multiple issue, longer exposure times sufferfrom a 1/f amplitude response, limiting the amount of information thatcan be encoded in one frame. In the case that the source is dim enoughto demand a longer integration, and if amplitude-only modulation isperformed at the source, the PSR demodulation outputs from multipleshort-exposure frames may be accumulated, improving the signal-to-noiseratio (SNR) and allowing more information to be decoded reliably.

FIG. 3A is a diagram 300 relating SNR to modulation frequency for asystem in which the exposure period is set to 1 ms and the frequency ofa single-sinusoid modulation function is varied over a range of fromzero to about 10 kHz. At integer multiples of 1 kHz (corresponding tothe reciprocal of the 1 ms exposure time), the SNR approaches zerobecause the exposure interval captures an integer number of cycles ofthe modulation. Demodulating a PSR provides a DC value of 50% of fullintensity. At frequencies that are half-integer (M+0.5) multiples of 1kHz, the 1 ms exposure time collects M full cycles of modulation (whichcancel to DC) plus one half-cycle that remains to vary the brightness ofthe PSF row-by-row. As M increases, that one half-cycle that remainscomprises about 1/(2*M+1) of the total modulation energy, explaining the1/f envelope (curve 305). In this example the SNR exceeds 10 out pastM=9.

FIG. 3B is a diagram 310 illustrating SNR as a function of exposure timefor a single PSR modulated using 3 kHz and 4.25 kHz sinusoids. Manyexposure times effectively cancel one of the two PSRs, potentiallysacrificing the ability to distinguish the point source responsible forthe PSR from another. By modulating with the sum of the 3 kHz and 4.25kHz sinusoids, only a single null 315 appears where both signals exhibitnulls. In some embodiments a coincident null occurs at the “temporalduration” of the PSR, defined here as the time of array 107 multipliedby the spatial height of the PSR in pixels. An imaging device or systemmay be programmed or otherwise configured to avoid exposure settingscorresponding to multiples of the temporal duration.

FIG. 4A is a cut-away view of an imaging image sensor 100 of FIG. 1A,like-identified elements being the same or similar. Grating 105 is abinary, phase-antisymmetric grating 105 overlying a CMOS (complementarymetal-oxide-semiconductor) array 107 of pixels 110, and may additionallyinclude a lenslet array that concentrates incident photons onto the mostsensitive areas of pixels 110 to increase quantum efficiency. Thefeatures of grating 105 offer considerable insensitivity to thewavelength of incident light in a wavelength band of interest, and alsoto the manufactured distance h between grating 105 and photodetectorarray 107.

Grating 105 produces an interference pattern for capture by array 107.Image information, such as one or more PSRs, can then be extracted fromthe pattern. Light in a wavelength band of interest strikes grating 105from a direction that is normal to the plane 400 of grating 105. Unlessotherwise stated, the wavelength band of interest is the near-infraredspectrum. Image sensors developed for use in different applications canhave different bands of interest, as is well understood by those ofskill in the art.

Grating 105 is formed by an interface between light-transmissive mediaof different refractive indices, an optical Lanthanum dense flint glasslayer 402 and polycarbonate plastic layer 405 above grating 105 in thisexample. Each of three boundaries of odd symmetry 410 is indicated usinga vertical, dashed line. The higher features 420 of grating 105 inducephase retardations of half of one wavelength (π radians) relative tolower features 415. Features on either side of each boundary exhibit oddsymmetry. With this arrangement, paired features induce respective phasedelays that differ by approximately half a wavelength over thewavelength band of interest (e.g., near-infrared light). Due todispersion, the difference in the refractive index of the Lanthanumdense flint glass layer 115 and the polycarbonate above grating 105 isan increasing function of wavelength, facilitating a wider wavelengthband of interest over which the phase delay is approximately π radians.These elements produce an interference pattern for capture by array 107.

Image sensor 100 includes an optional opaque layer 440 patterned toinclude an aperture that encompasses or defines the effective limits ofgrating 105. The aperture windows captured interference patterns, whichtends to reduce edge effects that result from subsequent image-recoveryalgorithms. The aperture can also improve angle sensitivity and spuriouslight rejection, which can be advantageous for e.g. motion detection andmeasurement. Opaque layer 440 can be applied directly to a layer forminggrating 105, and may be coplanar or nearly coplanar with grating 105.Other embodiments omit the aperture, or may include an aperture spacedaway from image sensor 100 instead of or in addition to the aperture inlayer 440.

The example of FIG. 4A assumes light incident the light interface ofimage sensor 100 is normal to the plane of phase grating 105, in whichcase, by Huygens' principle, pairs of spherical wave re-radiatorsequidistant from one of the boundaries of odd symmetry 410 cancel eachother out due to the half-wavelength phase delay of the radiator on oneside of the boundary 125 compared to the other. Thus, light of anywavelength in the band of interest destructively interferes to producecurtains of minimum intensity that extend to array 107 beneathboundaries 410. Neither the depth nor the wavelength of light over asubstantial spectrum significantly influences this destructiveinterference. Constructive interference similarly produces foci ofmaximum intensity that extend to array 107. Both the low and highfeatures 415 and 420 admit light, which provides relatively high quantumefficiency relative to embodiments that selectively block light.

FIG. 4B is a plan view of imaging image sensor 100 of FIG. 1A inaccordance with an embodiment in which grating 105 includes spiralfeatures 430 and 435 to produce two-dimensional diffraction patterns.Relatively narrow (wide) segment spacing works better for relativelyhigh (low) frequencies, feature spacing increases along odd-symmetryboundaries (between elevated and recessed grating regions, representedby dark and light) with distance from the center. Curved boundaries ofodd symmetry, defined between the elevated and recessed regions, extendradially from the center of the grating to the periphery, radiating outbetween the dark (elevated) and light (recessed) arms near the center.In some embodiments, the functional form of the curved boundariesapproximates a logarithmic spiral. The area of grating 105 can begreater than that of the aperture in layer 440 to provide alignmenttolerance in manufacturing.

FIG. 5 illustrates a system 500 in which a smart image sensor 505monitors the locations of some number of autonomous aerial vehicles(UAVs) 510 that emit from uniquely modulated illumination sources.Sensor 505 functions in the manner of sensor 103 of FIG. 1A todistinguish UAVs 510. Assuming a system where roughly sixty bits ofinformation are available in the way a PSF is modulated, confusing onedrone with another via a cryptographic “birthday attack” would start tobe expected only where at least 2{circumflex over ( )}30 (more than abillion) drones are simultaneously present.

The depth of modulation seen at the pixel array depends on exposure timeand modulation frequency. Longer exposures and higher frequenciesgenerally decrease modulation depth and limit the bandwidth available todistinguish and communicate via point sources.

Returning to the example of FIG. 1A, image sensor 100 does not require alens to produce images. Rather than focusing, as would be done by atraditional camera, image sensor 100 captures a diffraction pattern thatbears little resemblance to an imaged scene, but that is neverthelessinterpretable by a computer or processor. Grating 105 exhibits a PSFthat produces a multi-armed spiral PSR on array 107 for every point oflight in the imaged scene. The location of the center of a given PSR isuniquely determined by the incident angle of light from thecorresponding point source. Since faraway scenes can be thought of ascollections of point sources of varying intensity, the sensed PSRsresemble a convolution of the PSF with the faraway scene. Embodiments ofphase grating 105 and related elements are detailed in U.S. PatentPublication 2016/0241799 to Patrick R. Gill, which is incorporatedherein in its entirety.

Imaging systems of the type detailed herein have many uses. In atoll-payment application, for example, a vehicle or driver could arrangeto have a toll payment made at a certain geographic location. Part of asecure transaction could be the agreement on roughly 60 digits of aone-time-use code. When approaching the toll location, the vehicle couldthen flash either a specific light of a specific wavelength or perhapsits headlights or other exiting light with a modulation that encodes theshared secret one-time-use code. The toll imaging hardware then knowsthat this specific vehicle has paid their toll, and can track thecleared vehicle visually. Other nearby cars not displaying an authenticcode could be directed aside for secondary payment. This technologycould be much faster than existing RF transactions, which requirevehicles to slow down in part so the much longer-wavelength RFcommunications are sure to localize the correct cars to ensure thecorrect vehicles are permitted through the toll booth.

Bus headlights could encode their route numbers or other identifiers,allowing wearables to direct a user on the right routes with minimalattention. Other in-building and in-city navigation could be facilitatedby LED beacons broadcasting information about their location.

Indoor supplements to GPS signaling could also be implemented. An 80-bitsignal is sufficient to specify 31 bits of latitude and longitude plus18 bits of altitude: specificity to within about an inch. (Finerprecisions are made moot in a matter of years due to continental drift.)Wearables navigating by these beacons could allow location services on amuch finer scale than GPS, without any satellite receiver needed.

Authentication codes with spatial specificity can also be useful ine-commerce. For example, suppose a consumer pays for a physical objector service in a situation where several nearby consumers also want thesame thing. If they have a device capable of modulating a one-time-useconfirmation authenticating them as having payed, then selling hardwarecan pinpoint their location and deliver the goods or servicesautomatically to the right location. If near-field communication (NFC)is more cumbersome than a user in a checkout line would desire, and theuser trusts that no hackers have put malicious LEDs into the ceiling ofa store, then the user can use their smartphone to confirm a certainlow-bandwidth signal is authentic to the store. The low-bandwidth signalcould be the equivalent of a URL specifying an https website or someother identifier of a form of initiating a digital transaction with therightful owner of the space, using standard public key cryptography. Thecombination of the consumer smart image sensor and their accelerometercan distinguish the signal on the ceiling from any other nearby falselight sources, reducing the risk of a spoofing attack and providing aspatially vetted authentication signal beyond what is present in NFCpayment.

A smart optical sensor of the type detailed herein may be mounted with acoaxial focusing camera, and the user could be presented with areal-time video view of the scene with icon overlays on the detectedpoint sources. Tapping the desired one could trigger any of a number ofactions, for example ordering at a sushi restaurant. Each display itemhas a beacon, and the customer points their phone at the display andtaps on the ones they want. The beacons can also identify where thecustomer is in the store, allowing for example accurate delivery ofsushi to the correct table.

Tracking authenticated humans can also be made easier by having eachhuman tagged with a specific transponder code flashing either a fixedpattern or some form of encrypted signal. For example, once a secureconnection between the user's badge and a base station is first made(possibly over RF), the two parties can securely agree on a session keythat is then hashed with the current time each second, and a few digitsof this hash is flashed to the observing hardware every second. The useris then authenticated and their badge's position is monitoredcontinuously. Many users can be located to within a few arcminutes usingonly one small image sensor.

This scheme of hashed continuously changing modulation could be used inother scenarios as well, in place of the one-time-use codes. Othersimilar cryptographic methods for generating streamed symmetric cyphersare also great alternatives to the method described above where a hashof the current time plus a shared secret determines the ciphertext ofthe transponder.

A smart optical sensor can support low-latency vehicle-to-vehicle (orvehicle-to-city) communication by e.g. modulating existing vehiclelights or through dedicated wavelengths. The payload of a few bytescould serve merely as a transponder, tagging the visible locations ofcars with respect to each other, or could itself contain messagesregarding planned course corrections, upcoming hazards, etc.

Error correction codes or checksums may be used to increase theprobability of a correct transmission in any of the above scenarios.Where the message to be sent is slightly longer than the bandwidth of asingle frame, the message can be partitioned over a few frames of data.Synchrony between sender and receiver can help improve bandwidth andintegrity, although often it will not be necessary or easy to implement.

Angular velocity of a point source may also be estimated by geometricdistortion of the PSR captured from the pixel array. As rows are exposedin sequence, from top to bottom, horizontal motion of a point sourcewill result in the captured PSF being stretched diagonally, a distortionknown as “shear.” If the undistorted PSR fits neatly within a square,the distorted PSR will be fit best by a parallelogram whose left andright sides are not vertical and whose top and bottom edges arehorizontally displaced relative to each other. This distortion isstraightforward to estimate. Likewise, vertical motion of the pointsource will be apparent as a magnification of the PSR in the verticaldirection. As in the horizontal case, this magnification is due to thePSR being in different positions during the exposure times of differentrows. The row capturing the top edge of the PSR sees the PSR in aposition different from that seen by the row that captures the bottomedge of the PSR. Vertical motion may make a nominally 60-pixel tall PSFappear to be 55 or 65 pixels tall, according to its vertical velocity.

While the subject matter has been described in connection with specificembodiments, other embodiments are also envisioned. For example, thewavelength band of interest can be broader or narrower than those of theforegoing examples, and may be discontinuous. Disambiguation andthree-dimensional resolution can be enhanced by imaging point sourcesfrom multiple angles using multiple smart optical sensors of the typeddetailed herein. Other variations will be evident to those of skill inthe art. Therefore, the spirit and scope of the appended claims shouldnot be limited to the foregoing description. Only those claimsspecifically reciting “means for” or “step for” should be construed inthe manner required under the sixth paragraph of 35 U.S.C. § 112.

What is claimed is:
 1. An optical system to sense incident light, theoptical system comprising: an optic exhibiting a point-spread function,the optic to produce a point-spread response responsive to the incidentlight; rows of pixels, each pixel to sample a respective region of thepoint-spread response, each of the rows of pixels producing a row ofsample values, the rows of sample values providing an image frame; andat least one processor to: read the rows of sample values within theimage frame; and use a difference in relative intensity between thesuccessively read rows of sample values within the image frame toidentify modulation information.
 2. The optical system of claim 1,wherein the incident light comprises point sources, the at least oneprocessor to distinguish the point sources responsive to the modulationinformation.
 3. The optical system of claim 1, further comprising memoryto store a kernel representative of the point-spread function of theoptic.
 4. The optical system of claim 3, the at least one processor todetect a point-spread function from the rows of sample values and alignthe point-spread function with the kernel before comparing each of therows of sample values with a corresponding slice of the kernel.
 5. Theoptical system of claim 3, wherein demodulating the intensity valuescomprises comparing each of the rows of sample values to a correspondingslice of the kernel.
 6. The optical system of claim 3, the memory tostore a modulation signature, the at least one processor to compare themodulation information to the modulation signature.
 7. The opticalsystem of claim 6, the memory to store a second modulation signature,each of the first-mentioned and second modulation signaturescorresponding to a respective one of a first illumination source and asecond illumination source, the at least one processor to distinguishbetween the first illumination source and the second illumination sourceusing the modulation information.
 8. The optical system of claim 1,further comprising a modulated illumination source to produce at least aportion of the incident light.
 9. The optical system of claim 8, themodulated illumination source exhibiting a modulation period.
 10. Theoptical system of claim 9, the at least one processor to read eachsuccessive row a row time less than the modulation period after a priorrow, and to read all of the rows of sample values over a frame timegreater than the modulation period.
 11. The optical system of claim 9,further comprising a second modulated illumination source exhibiting asecond modulation period different from the first-mentioned modulationperiod.
 12. The optical system of claim 11, the at least one processorto read each successive row a row time less than the first-mentionedmodulation period and the second modulation period after a prior row,and to read all of the rows of sample values over a frame time greaterthan the first-mentioned modulation period and the second modulationperiod.
 13. The optical system of claim 1, the point-spread responseilluminating R of the pixels and defining a convex hull over S of thepixels, wherein S>2R.
 14. A method for identifying a modulated pointsource, the method comprising: exposing rows of pixels to a pattern fromthe modulated point source; sampling the pattern using the rows ofpixels; locating a point-spread response in the sampled pattern, thesampled pattern having a row of sample values for each of the rows ofpixels; accumulating an intensity value for each of the rows of samplevalues; and demodulating the accumulated intensity values.
 15. Themethod of claim 14, further comprising cropping the point-spreadresponse before accumulating the intensity values.
 16. The method ofclaim 14, wherein locating a point-spread response in the sampledpattern comprises correlating the sampled pattern with a kernelrepresentative of the point-spread response.
 17. The method of claim 16,further comprising storing the kernel.
 18. The method of claim 14,further comprising forming the pattern from light emanating from a scenethrough a phase grating.
 19. The method of claim 14, further comprisingsuccessively sampling the rows of pixels.
 20. The method of claim 19,wherein the modulated point source exhibits a modulation period, andwherein each successive row is sampled a row time less than themodulation period after a prior row.
 21. The method of claim 20, whereinall of the rows are sampled over a frame time greater than themodulation period.
 22. An optical system to sense incident light, theoptical system comprising: an optical grating exhibiting a point-spreadfunction, the optical grating to produce a diffractive response to theincident light; rows of pixels, each pixel to sample a respective regionof the diffractive response, each of the rows of pixels producing a rowof sample values; and means for a processor to use a difference inrelative intensity between successively read rows of sample valueswithin a frame to identify modulation information.